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2 VIDEO AND MULTIMEDIA BROWSING WHILE SWITCHING BETWEEN VIEWS 

3 

4 Technical Field 

5 The invention is in the field of video and multimedia browsing. More particularly, the 

6 invention relates to the ability to switch back and forth between different media streams within a 

7 browser environment. 

8 Background 

9 Computers have ushered in the Information Age: With their ability to manage and* 

10 process large amounts of data, the ability of the user to perform mundane tasks has been 

1 1 enhanced, and activities which were previously impossible are now commonplace. As 

12 computers have become more sophisticated, the tools and options at the disposal of the user have 

13 increased. For example, video browsing and, more generally, multimedia browsing are now 

14 commonly performed over the Internet or on local systems and networks such as a company's 

1 5 Intranet. Browsing may be conducted for any number of purposes, including entertainment, 

16 education (such as distance learning), or accessing the news. 

1 7 Information in written form, such as a book, generally includes a table of contents and/or 

1 8 an index which permits the user to quickly evaluate the book^s content and/or relevance to the 

19 user's interests. Browsing a book can be done quickly by simply flipping through the pages. 

20 Video browsing, on the other hand, while providing a number of advantages to the user, can be a 

2 1 very time consuming and frustrating process. Browsing a video to get an overview of its content 

22 is not generally time efficient. It may involve playing certain segments of the video and then fast 

23 forwarding or skipping to points of potential interest. This is especially time consuming in the 
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1 case of an analog recording medium such as a tape, but even with digital recording media, the 

2 user must still take the time to watch various video segments. 

3 To address these concerns, various techniques for summarizing video in a more compact 

4 form have been advanced. The use of "storyboards" is one such technique, in which a table of 

5 keyframes selected from the video are presented, possibly along with text or other "static" or 

6 stationary information. "Moving storyboards", or slide shows, are media streams which take the 

7 storyboard technique a step further by presenting a series of stationary images along with audio. 

8 "Fast playbacks", in which the video is fast forwarded, and "video summaries" ("video skim"), in 

9 which key video segments are presented, are two other video visualization technologies for more 

10 compactly presenting audiovisual information. "Talk presentation", in which video, audio, and 

1 1 speaker's slides play together in one or more windows, is another visualization technology. 

1 2 These and other possible representations of the original video are referred to herein as different 

1 3 "views" of the video. 

14 US Patent 6,166,735 to Dom et al. ("Video story board user interface for selective 

1 5 downloading and displaying of desired portions of remote-stored video data objects") and the 

16 article "What is in that video anyway: in search of better browsing" (S. Srinivasan et al., 6th 

1 7 IEEE International Conference on Multimedia Computing and Systems, June 7-11,1 999, 

1 8 Florence, Italy) are concerned with switching from a (static) storyboard to the corresponding 

19 video. When the user selects a keyframe in the storyboard, the video starts to play from a point 

20 in the video corresponding to the selected frame. This technology relies on the use of static links 

2 1 with offsets, in which the offsets do not depend on when the user selects the keyframe of interest. 

22 Furthermore, with this technology, the user cannot switch back from a point in the video to a 

23 corresponding point in the storyboard (i.e., the linking is in one direction only). This method 
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1 does not allow switching from a first media stream to a second, related media stream beginning 

2 at a point in the second media stream which corresponds to the point in the first media stream 

3 where the first media stream was playing when the user made his or her selection. Thus, with 

4 this approach, the ability of the user to switch back and forth between a video and its related 

5 views is limited and not satisfactory. In particular, the user is frustrated since a compact view is 

6 not synchronized with its corresponding full length video or with other views. Thus, by way of 

7 example, switching from a point in a video summary to the corresponding point in the full video 

8 is currently not supported. What is needed is a system that provides a way of directly switching 

9 between different views, or more generally between different media streams, such that the point 

10 at which a media stream is entered corresponds to the point at which the previous media stream 

1 1 is exited. 
12 

13 Summary of the Invention 

14 Preferred implementations of the invention permit a user to seamlessly switch from a first 

15 media stream to a second media stream in a synchronized way, such that the second media 

16 stream picks up where the first media stream left off. In this way, the user experiences events 

1 7 chronologically without interruption. In a preferred implementation, a user watching a skim 

1 8 video (a compact, edited version of a video which is shorter in length than the full video and 

19 conveys a summary of the full video) switches to a full length video when, for example, the skim 

20 video reaches a frame that is of particular interest to the user. The full length video begins at or 

2 1 near a point corresponding to the frame in the skim video that is of interest to the user, so that the 

22 user does not experience any time gaps in the story line. An advantage of preferred 

23 implementations described herein is that the user decides which media streams to select. 
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1 One implementation is a method of browsing, which includes accessing a first media 

2 stream and allowing the first media stream to play up to some point in the first media stream 

3 The method further includes switching media streams by selecting a second media stream 

4 different from the first media stream, which thereby starts the second media stream at a point 

5 corresponding to said point in the first media stream. In a preferred implementation, the second 

6 media stream has media content in common with the first media stream. One of the media 

7 streams may be selected from the group consisting of skim video, moving storyboard, full video, 

8 audio, text, animation, and graphics. In one preferred implementation, the point in the second 

9 media stream is selected from one of a plurality of points in the second media stream 

1 0 corresponding to the first media stream. 

1 1 In yet another preferred implementation, the point in the second media stream 

12 corresponds in the chronological sense to said point in the first media stream. The first media 

13 stream may include a skim video, and the second media stream may include a full video. The 

14 starting point in the second media stream may advantageously compensate for user response time 

1 5 and thus not be precisely synchronized with the point in the first media stream. Alternatively, the 

16 starting point in the second media stream may advantageously correspond to the beginning of a 

1 7 shot, a video scene, or a sentence (e.g., a spoken sentence or a sentence appearing as part of 

1 8 closed caption text). 

19 Another implementation is a method of providing media streams to a browser user, which 

20 includes providing a first media stream and receiving a request from the user to access a second 

21 media stream different from the first media stream, in which the request is generated at some 

22 point in the first media stream. The method further includes providing, as a result of the request, 

23 the second media stream from a point corresponding to said point in the first media stream. In a 



IBM 92001 0062US1 

1 preferred implementation of the method, the second media stream has media content in common 

2 with the first media stream. In another preferred implementation, said point in the second media 

3 stream corresponds in the chronological sense to said point in the first media stream. The 

4 starting point in the second media stream may advantageously compensate for user response time 

5 and thus not be precisely synchronized with said point in the first media stream. Alternatively, 

6 said starting point in the second media stream may advantageously correspond to the beginning 

7 of a shot, a video scene, or the beginning of a sentence (e.g., a spoken sentence or a sentence 

8 appearing as part of closed caption text). In one implementation, said point in the second media 

9 stream is selected from one of a plurality of points in the second media stream corresponding to 
10 the first media stream. 

^0 
CO 

r y 1 1 In other implementations, there are provided computer program products for carrying out 



□ 1 2 any of the methods herein. 

W 

Irt 13 Implementations of the invention are disclosed herein in which the user's experience is 



ru 



14 enhanced by enabling him or her to switch between different media streams in a synchronized or 



ij 1 5 nearly synchronized fashion to reduce the time spent searching for particular video segments of 

a 

$a 16 interest. These implementations provide the user with video browsing techniques which are 

1 7 more powerful but also more flexible than those in the prior art. 

18 In a preferred implementation, Real Media is used for the streaming, whereas Real Player 

1 9 is used as the plug-in player for streaming video and is embedded in a web browser such as 

20 Netscape Navigator or Microsoft Internet Explorer. The program logic is implemented in 

2 1 Javascript code embedded in an HTML page sent from the HTTP server to the client. The 

22 browser may advantageously be used as the front end for the IBM CueVideo content-based video 

23 and audio retrieval system. This implementation is particularly well suited to on-line browsing 
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1 of videotaped conferences, allowing users to search, retrieve, and browse particular video 

2 segments of interest from a large collection of video clips. 
3 

4 Brief Description of the Figures 

5 Figure 1 shows a computer monitor screen and a viewing panel or window for watching 

6 videos, illustrating one implementation of the invention. 

7 Figure 2 illustrates timelines showing how a preferred implementation may be used in 

8 practice. 

9 Figure 3 shows a graphical user interface (GUI) on the screen of a monitor for watching 

10 videos, illustrating a preferred implementation of the invention. 

1 1 Figures 4, 5, and 6, which includes Figures 6A and 6B, are schematic diagrams 



C3 12 illustrating the logic used in certain implementations of the invention. 
U 

m 



13 Figure 7, which includes Figures 7A-7H, is a section of computer code showing 

[y 14 JavaScript functions for synchronous playback which may be used in a preferred implementation. 

15 Figure 8 is a schematic diagram of an architecture for supporting preferred 

16 implementations. 

1 7 Figures 9 and 10 are look-up tables which facilitate synchronous switching from one 

1 8 media stream to another media stream. 
19 

20 Detailed Description of the Invention 

2 1 Preferred implementations of the invention provide the user with the ability to seamlessly 

22 switch, without loss of context, between different media streams related to each other (e.g., they 

23 may have media content in common with each other, or they may correspond to each other in the 
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1 chronological sense), but which may have different media tracks, media types, or time scales. A 

2 preferred implementation provides a user the ability to switch back and forth in a synchronized 

3 way between one media stream (e.g., a skim video) to a second media stream (e.g., the foil or 

4 complete video). Other kinds of media streams to be used in other implementations include but 

5 are not limited to audio, sped-up audio in combination with video (such as that described in 

6 "Using audio time scale modification for video browsing", A. Amir et al., Thirty-third Hawaii 

7 International Conference on System Sciences, HICSS-33, Maui, January 2000), skim video, and 

8 closed caption text. Switching between media streams directed to different senses (e.g., between 

9 an audio stream and a video stream) is also contemplated. The synchronization between media 

1 0 streams may be implemented and coordinated either at the server end or the client end. The 

1 1 switching between streams is preferably based on simple user input, such as the click of a mouse 

12 or depressing a key on a keyboard, thereby permitting the user to focus on the results generated 

13 by the switching of media streams, rather than on the mechanics of switching process. 

14 For example, the user can watch a skim video or a moving storyboard, and then select the 

1 5 complete video stream when the user sees a particular frame of interest. The selected video then 

1 6 begins at (or near) that point in the video corresponding to the frame of interest. This saves the 

1 7 user the trouble of having to either manually locate that portion of the video of interest or watch 

1 8 the entire video. The user may switch from one media stream to another and back again by 

1 9 clicking a mouse button or a keyboard key. These and other implementations are particularly 

20 well suited for integration into the IBM Cue Video technology, as described in U.S. Patent 

2 1 6,1 66,735 to Dom et al. entitled "Video story board user interface for selective downloading and 

22 displaying of desired portions of remote-stored video data objects"; U.S. Patent 6,185,527 to 

23 Petkovic et al. entitled "System and method for automatic audio content analysis for word 
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1 spotting, indexing, classification and retrieval"; U.S. Application No. 09/572136 to Amir et al. 

2 entitled "Fast video playback with automatic content-based variable speed" filed May 16, 2000; 

3 U.S. Application No. 09/028754 to Dom et al. entitled "Method and system for automatic 

4 computation of a video storyboard" filed February 24, 1998; U.S. Application 09/295076 to Amir 

5 et al. entitled "System and method for linking an audio stream with accompanying text material" 

6 filed April 20, 1999; and "Towards automatic real time preparation of on-line video proceedings 

7 for conference talks and presentations" by A. Amir et al. (Proceedings of the 34th Hawaii 

8 International Conference on System Sciences, HICSS-34, Maui, January 2001), all of which are 

9 hereby incorporated by reference. 

.510 By way of example, and with respect to Figure 1 , a video monitor screen 1 0 is shown on 

CO 

jy 1 1 which a window 14 is displayed. Other windows and information may be displayed on the 

Q 12 monitor screen 10 as well, as discussed below in connection with Figure 3. The window 14 

t : J t t 

l ? ! 13 might display a first, compact media stream such as a skim video. Alternatively, another ' 

y 14 compact media stream such as a fast playback or a moving storyboard may be displayed. The 

fll 

*Q 1 5 user switches to a second media stream at some point, in which the first media stream and the 

a 

& 16 second media stream have media content in common with each other. The second media stream 

1 7 could be the full length (i.e., original, unedited) video, for example. 

1 8 The user begins by observing the video skim, thereby quickly obtaining an overview of 

19 the corresponding full length video. If a particular frame or scene within the video skim is of 

20 interest to the user, the user may then click on a button or on a mouse (not shown) to start the 

21 second media stream, i.e., the full length video. In particular, the user may place a cursor 

22 controlled by the mouse on a tab 1 8 and click when the first media stream (displayed within the 

23 window 14) reaches a point that the user wishes to observe in more detail. The full length video 

8 
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1 then begins at (or near) a point in the video determined by that point in the video skim when the 

2 user registered his or her preference to stop the video skim and switch to the full length video. 

3 For example, if the user observes in the video skim that a professor is moving to a podium to 

4 deliver a lecture, and the user desires to watch the lecture in full video mode, the user can click 

5 on the mouse, so that the full length video begins with the professor moving to the podium. In 

6 this manner, switching from the first media stream (the video skim) to the second media stream 

7 (the full length video) is synchronized, sparing the user the inconvenience of having to watch 

8 those portions of the full length video that are not of interest. 

9 This process is illustrated by the example shown in Figure 2. Time lines 20 and 2 1 
10 correspond to a first media stream and a second media stream, respectively, which, without loss 

f y 1 1 of generality, may be a video skim and its corresponding full length video. The video skim is 

Q 12 ideally a condensed version of the full length video in which the frames are chosen to include key 

iu 

l? 1 13 events within the full length video. A user accesses the video skim at a frame corresponding to 

£3 

Q 14 point 22 and continues to view frames (e.g., at points 22a, 22b) until some later frame 

fcjj 1 5 corresponding to point 23. Point 22 may correspond to the start of a soccer game, for example, 

£3 

1 6 and point 23 may correspond to the scoring of the first goal. At point 23, the user selects the full 

1 7 length video corresponding to the time line 2 1 . The user's viewing window (e.g., the window 14 

1 8 in Figure 1 ) is then switched to the frame in the full length video showing the scoring of the first 

19 goal. This frame in the full length video is denoted by the point 23' along the full video's time 

20 line 2 1 . The user then watches the full length video up to a frame denoted by the point 24' (e.g., 

2 1 the scoring of the second goal), at which point the user switches back into the skim video 

22 representation at a frame denoted by the point 24. The user may then observe additional frames, 
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1 such as the one denoted by the point 25. (Frames which were not accessed by the user in this 

2 example are denoted by the open circles 26.) 

3 The pair of points 24 and 24' thus represent the same point in the chronological sense (in 

4 this example, the scoring of the second goal), just as do the pair of points 23 and 23' (in this 

5 example, the scoring of the first goal). However, it may be advantageous to introduce some 

6 small differences in chronological time when switching from one media stream to another media 

7 stream: it may be desirable, for example, that when the user selects the full length video as the 

8 skim video reaches the scoring of the first goal, that the user is brought to a frame corresponding 

9 to a point in time a few seconds before the scoring of the first goal (e.g., less than about 2, 3 or 4 

1 0 seconds), to compensate for user response time and to allow for an appropriate lead-in. 

1 1 (Alternatively, the starting frame may default to the beginning of the shot containing the 

12 corresponding point in time of interest.) This starting point technique is useful when moving 

1 3 from a long media stream to a shorter media stream, so that the starting point in the shorter media 

14 stream may be set to the beginning of the last event included in the video skim-the starting 

1 5 frame in the second media stream is nevertheless determined by (and corresponds in the 

16 chronological sense with) the exit frame in the first media stream, even though there is not exact 

1 7 synchronization in this case. Likewise, if the full length video reaches the frame corresponding 

18 to the point 24' (the scoring of the second goal) and the user selects the skim video representation 

1 9 a short time (e.g., a few seconds) after the second goal is scored, it may be advantageous to bring 

20 the user to the frame showing the scoring of the second goal. Additionally, in preferred 

2 1 implementations, the user may adjust the player position along the media stream time line 

22 manually at will (e.g., after being switched to a different media stream), to fine tune his or her 
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1 position within the selected media stream. This is discussed below in connection with the time 

2 scroll tab of Figure 3. 

3 Although implementations have been described above with respect to skim video as a 

4 first media stream and a full length video as a second media stream, other implementations 

5 present the user the option of accessing several different media streams having media content in 

6 common with each other (e.g., the different media streams correspond to several different views). 

7 In addition, with other implementations, the user begins by first accessing a database of many 

8 different media clips which do not necessarily have media content in common with each other 

9 (e.g., the media clips may be altogether different video segments) and then selecting one of them. 
£3 1 0 One such implementation is described with reference to Figure 3, in which tabs are displayed on 

jiy 11a video monitor screen 60 for a (non-moving) storyboard 62 as well as the following media 

ry 

□ 12 streams: moving storyboard 64, fast moving storyboard 68, slow moving storyboard 72, and full 
W 

Ifi 1 3 (length) video 76. In the implementation shown in Figure 3, the user begins by searching a 

W 14 database of topics, e.g., by entering key words into a search engine (not shown in Figure 3). 

HI 

jj 1 5 Once the search engine identifies media clips of interest, the user selects one of them, resulting in 
□ 

& 1 6 the selected media clip starting to play at a point related to the user's search criteria. This media 

1 7 clip is then displayed within a viewing window 78 on the video monitor screen 60, as shown in 

1 8 Figure 3. The view of the media clip that is displayed can be the full video, or a particular view 

19 (e.g., skim video) that has been preselected by the user as the default view. 

20 The user then selects one of the tabs displayed on the screen 60, e.g., the moving 

2 1 storyboard tab 64. At any point during the playback of the moving storyboard presentation, the 

22 user may switch from this media stream to another media stream by selecting one of the tabs 68, 

23 72, or 76. By switching back and forth between these tabs, the user selects the rate at which he 

11 
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1 or she receives data, but does not access an unrelated media clip. The user may also fine tune his 

2 or her position within a selected media stream by moving a time scroll tab 86. 

3 Additionally, the user may select a different media clip by selecting one of the result tabs: 

4 Previous Result 80 or Next Result 84. Selecting one of the result tabs brings the user to a point 

5 in another media clip found by the search engine, and this other media clip may or may not have 

6 media content in common with the currently running media stream. 

7 In addition to the video representations discussed above, implementations may include 

8 other video representations such as full video at various speeds (such as fast playback or slow 

9 motion), a full video at a fixed or variable speed determined by the user, a reverse playback (e.g., 

n 

| 10 fast reverse using the Apple QuickTime Player, as described in U.S. Patent 5,267,334 to 

CO 

fU 1 1 Normille et al. entitled "Encoding/decoding moving images with forward and backward 

m 

Q 1 2 keyframes for forward and reverse display" and U.S. Patent 5,739,862 to Cen entitled "Reverse 

W 

* n 1 3 playback of MPEG video"), a low bandwidth (e.g., 50 kbps) representation of the original, a 

p 

Q 14 panoramic video (see, for example, "Enhancing distance learning with panoramic video", by J. 

ru 

• % q 1 5 Foote and D. Kimber, Proceedings of the 34th Hawaii International Conference on System 
Q 

^ 16 Sciences-2001), a video preview, and a video trailer. "Mosaic" video composed of multiple 

1 7 video tracks can be employed in which either moving storyboards or other kinds of video are 

18 employed. In the case of mosaic video, some views may be dedicated to a global view (e.g., an 

1 9 entire soccer field), whereas other views may focus in on areas of particular interest (e.g., an 

20 individual player on that field). Other media streams may include animation (such as fast 

2 1 moving storyboard without audio) and graphic display (e.g., 3-D graphics). In general, a media 

22 stream may be composed of one or more tracks playing in parallel in a synchronous way as a 

12 
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1 multi-track media stream (as described in, for example, U.S. Patent 5,452,435 to Malouf et al. 

2 entitled "Synchronized clocks and media players"). 



4 audio track which corresponds to the full video may be offered to the user as one of the tab 

5 options on the video monitor screen 60. The audio may be streamed out at anywhere from 

6 normal speed to up to, say, 2, 5 or even 10 times normal speed. An audio track in a language 

7 other than the language originally recorded may be used; e.g., a speaker may have his or her 

8 words translated into other languages, with the translated words being placed on a separate audio 

9 track. This permits the user to switch from an audio stream to a video stream, or vice versa. 

10 Alternatively, a dubbed sound track with video may be offered as an option. In the case of a 

1 1 video being played faster than normal, the audio component can be a sped-up audio or a 

12 summary of what was said, with the summary being presented at a normal or fast spoken rate and 

13 with the length of the spoken summary corresponding to the length of the video summary, 

14 Other implementations may include text as a media stream, e.g., closed caption text. 

15 Closed caption text may include the original language or the original words translated into any 

16 language. If closed caption text is used in a fast forward mode, a larger font for key words or 

1 7 phrases can be used so that the user can appreciate the most important topics at a glance. In the 

1 8 case of a technical talk, one media stream can include a high quality image of any slides being 

19 presented. In the case of a lecture, the streaming text could be a transcript of what is being said 

20 or parts thereof, a list of the topics or phrases, or a table of contents related to the talk. Other 

2 1 implementations may include hyperlinks which are embedded in certain media streams, e.g., as 

22 "hotlinks" in a video stream (see, for example, U.S. Patent 6, 1 75,840 to Chen et al. entitled 



23 "Method for indicating the location of video hot links" and U.S. Patent 6,137,484 to Hoddie et al. 

13 
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1 entitled "Method and apparatus for identifying user-selectable regions within multiple display 

2 frames") so that the user can be directed to various web pages of potential interest. Other 

3 implementations may have a window tied to a media stream which provides sign language. 



5 implementations like that shown in Figure 3. Figure 4 outlines logic for switching from a point 

6 in a first media stream to a point in a second media stream corresponding to the point in the first 

7 media stream where the first media stream was exited. A first media stream is accessed by the 

8 user in step 90, and the user observes this first media stream. At some point in the first media 

9 stream denoted by step 92, the user decides to switch to a second media stream having media 

10 content in common with the selected media clip, such as those represented by the tabs 64 

1 1 (moving storyboard), 68 (fast moving storyboard), 72 (slow moving storyboard), and 76 (full 

12 video). This point in the first media stream corresponds to point 23 in the example shown in 

1 3 Figure 2 above. The current position in the first media stream is determined (step 94), the 

14 corresponding position in the second stream is calculated (step 96), and the playback in the 

1 5 current view is stopped (step 98). The playback is then started at the calculated point in the 

16 second media stream (step 101). This calculated point in the second media stream corresponds to 

1 7 point 23 ' in Figure 2 in the example above. (Alternatively, the playback in the current view may 

1 8 be stopped (step 98) before the starting position in the new view is determined (step 96). As a 

1 9 further alternative, the first media stream may continue to play until after the second media 

20 stream has started (step 1 0 1 ), or the actual switching can be done using standard video editing 

2 1 transition effects, such as a fade or a dissolve.) In a like manner, other views may be accessed by 

22 the user by repeating the algorithm. 



4 



Flow charts are now presented which describe the logic used in algorithms for 
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1 Figure 5 outlines logic which builds on the logic illustrated in Figure 4, by considering a 

2 preliminary step of generating a list of media clips to browse, in which the media clips do not 

3 necessarily have media content in common with each other. In this implementation the list is 

4 generated from a search as discussed above in connection with Figure 3. (Alternatively, the user 

5 may simply select a media clip from a list of media clips, for example.) Thus, the user may 

6 advantageously switch between different views related to a single result from the search, or 

7 switch from one search result to another search result, and the system ideally differentiates 

8 between these situations. As a result of the search, a list is generated (step 102), which 

9 preferably includes an index of media clips as well as one (or more) preferred starting points 

s ? 1 0 within each of the media clips. These starting points ideally correspond to points within the 

^9 
in 

f2 1 1 media clips which relate most closely to the user's topic of interest. Thus, the search results may 

ru 

£3 1 2 be thought of as a table of paired parameters: the media clips found by the search and respective 

W 1 3 starting points (offsets) within each of those media clips. The user accesses a first media stream 

1 4 (step 90), such that the corresponding full video (or alternatively a related view (e.g., skim 

ry 

1 5 video), which may have been preselected by the user as the default view) is displayed. At some 

5 

fa 1 6 later point, the user selects a second media stream (step 92), e.g., by clicking on a button in a 

1 7 graphical user interface (GUI). If the user selects a different view, such as a Fast Moving 

1 8 Storyboard, the decision in step 1 05 is YES, and the algorithm proceeds to determine the current 

1 9 position in the first media stream as described in connection with Figure 4. 

20 If, however, the user has elected to switch to another search result (e.g., by pressing the 

21 Next Result tab 84 shown in Figure 3), then the decision in step 105 is NO. In step 106, the 

22 algorithm retrieves the starting position in the media stream corresponding to this other search 

23 result by using the list generated in step 102. In this case, there is no need to determine the 

15 



IBM 920010062US1 

1 current position in the first media stream, since the first media stream and the second media 

2 stream do not necessarily have media content in common with each other. The user may 

3 subsequently select other media streams to view, thereby repeating the algorithm or a portion of 
- 4 it. 

5 In certain implementations, there may be more than one point in the second media stream 

6 corresponding to the point in the first media stream at which the second media stream is selected. 

7 In this case the second media stream and the first media stream may have media content in 

8 common with each other, but the points in the second media stream may not necessarily 

9 correspond in the chronological sense with the point in the first media stream. For example, the 
fc jjSj 10 first media stream could be a video summary of one particular news story, and the second media 

f <s 

ry 1 1 stream could be the last 24 hours of broadcast news on a cable TV station. The story and updated 

m 

C3 12 versions of it could have been broadcast at several different times during this 24 hour period. In 

W 

W 13 this case, the user may wish to see some or all of the different versions of the story which were 

5 

H 

j"* 14 broadcast; each of these versions has a corresponding starting point in the second media stream. 

yjj 15 A list of all these starting points can be kept as a table and can be accessed in various ways. In 

5 

j=* 1 6 one implementation, the second media stream starts to play at a point corresponding to the first 

1 7 broadcast of the story, and the user can press a button "Next Result' 1 to skip to the next broadcast 

1 8 of the story. In another implementation, a list of several or all of the points corresponding to the 

1 9 various broadcasts of the story can be displayed, and the user selects which one of these 

20 broadcasts he or she would like to play. In yet another implementation, a play-list, composed of 

2 1 several or all of the video segments related to the story of interest is played. In this case, the user 

22 watches all these segments as if they were combined together into one continuous video. 
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1 Figure 6 is directed to an implementation related to the IBM Cue Video Toolkit, in which 

2 an application makes use of SMIL files with Real Player. The switching from the first media 

3 stream to the second media stream in this case involves additional steps. Because of a limitation 

4 in the current Real Player Application Program Interface (API), the position in the second media 

5 stream, when it is a stream composed using a SMIL file, cannot be set before the second media 

6 stream has already started to play. After the first media stream has been stopped (step 98), the 

7 Real Player is set to the second media stream (step 130). In step 132, the Real Player waits for 

8 the second media stream to fill the buffer in the Real Player and starts to play the second media 

9 stream from its beginning. The Real Player then pauses the playback (step 134), sets the Real 

t P 1 0 Player position in the second media stream to the calculated position (step 1 36), in which the 

SO 

ry 1 1 calculated position has been calculated in step 96. The Real Player then begins playing the 

ru 

O 12 second media stream at this calculated position (step 138). The logic inherent in Figure 6 is 
W 

^ 1 3 otherwise similar to that described above in connection with Figures 4 and 5. The user may 

□ 

[y 14 subsequently select other media streams to view, and the algorithm or a portion of it may be 

ru 

4$ 15 repeated. 
£3 

fa 1 6 Figure 7 gives JavaScript code, embedded in a DHTML page, which uses the API of the 

1 7 Real Player to implement the functionality described in connection with Figure 6. These 

1 8 JavaScript functions, which are designed for the client side, provide synchronous playback based 

19 on asynchronous streaming video APIs. The different media views (such as those shown in 

20 Figure 3) are stored as files on the video server 198, whereas, in this implementation, the browser 

2 1 is implemented as an HTML page that contains Javascript code and an embedded Real Player 

22 object. Although a preferred implementation has been described with respect to Real Player, 
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1 other implementations involve the use of players other than Real Player and may be used with a 

2 variety of video, audio and media players. 

3 A preferred architecture for the implementations disclosed herein is now described, which 

4 may be used in either Internet or Intranet applications. Figure 8 illustrates architecture for 

5 supporting two or three-tier web applications in the context of media streaming. This 

6 architecture supports the searching of a media (e.g., video) database by typing in keywords, and 

7 retrieving and playing back media clips selected by the user. The user begins with a standard 

8 "search" page 1 70 in Hyper-text markup language (HTML) presented by the client's browser (the 

9 Netscape Navigator or Microsoft Internet Explorer platforms may be used). The page 1 70 
10 contains a POST form 1 75 for typing in keywords, which the user then submits by clicking on a 

m 

j y 1 1 "Submit" button. This results in an HTTP request 1 78 being sent to a web server 1 82, which in 

jfi z 
5 

□ 1 2 turn calls up an application server 1 86 . The application server 1 86 performs the search, and 

*^ 1 3 generates the search results as an HTML page 1 90. The user then selects one or more of the 

C3 

s s ; | 14 media clips found by the search (each of which preferably has an associated time offset as 

ru 

1 5 discussed above in connection with Figure 5), and the media browser allows the user to choose 

Q 

16 from a number of representations of a given media stream, such as those indicated in Figure 3. 

1 7 The client can play a desired video clip at the appropriate time offset by communicating through 

1 8 a streaming video player plug-in 1 94 (such as Real Plugin or the IBM Video Charger Plugin, 

19 installed at the client side) to a streaming video server 198 (e.g., Real Server or the IBM Video 

20 Charger Server). Details regarding the IBM Video Charger Plugin and the IBM Video Charger 

21 Server may be found at http://www-4.ibm.com/software/data/videocharger/ . However, 

22 implementations of the invention may be used with a variety of media streaming formats, servers, 

23 browsers, and players. The graphical user interface (GUI) functionality used to support playback 
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1 based on query results is thus extended to support the switching of views from a first media 

2 stream (e.g., skim video) to a second media stream (e.g., full length video), starting with the 

3 appropriate time offset given by the first media stream at the time of switching. 

4 Although the video browsing content can be precomputed and streamed from the video 

5 server 198, video control logic (such as calculating time offsets and naming appropriate video 

6 source files) may be advantageously generated by a Common Gateway Interface (CGI) program 

7 and encapsulated in Dynamic HTML (DHTML) at the client side, as shown in Figure 7. As is 

8 known in the art, CGI is a standard for external gateway programs which permits them to 

9 interface with information servers such as HTTP servers. Various hardware and software 

10 arrangements for implementing these procedures may be used. 

1 1 Various methods for calculating the starting points in a media stream are now described. 

12 In general, starting point calculations (offset calculations) may be performed on either the server 

13 side or the client side. In its simplest form, a mathematical function (e.g., a linear interpolation 

14 routine) may be employed to determine the starting point within a media stream. One starting 

15 point calculation algorithm which can be used in switching from a moving storyboard view to a 

16 full video view, as in implementations discussed herein, relies on a linear time correspondence 

1 7 calculation, i.e., the time in the second media stream is calculated by multiplying the time in the 

1 8 first stream by a constant, in which the constant is given by the relative speedup (compression) 

19 ratio between the two views. For example, if the compression ratio between a fast moving 

20 storyboard and its corresponding full video is 5, then 1 minute of viewing time in the fast moving 

2 1 storyboard corresponds (in the chronological sense) to 5 minutes of viewing time in the full 

22 video. In the case of nonlinear fast video playback, in which times depend on shot boundaries, 

23 the use of mathematical functions may be particularly advantageous in calculating the 
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1 appropriate viewing time or frame number in a desired new view, as described in U.S. 

2 Application No. 09/572 1 36 to Amir et al. entitled "Fast video playback with automatic 

3 content-based variable speed" filed May 16, 2000 and by S. Srinivasan et al. in "What is in that 

4 video anyway: in search of better browsing" (6th IEEE International Conference on Multimedia 

5 Computing and Systems, June 7-11, 1999, Florence, Italy) 

6 One straightforward method of managing time offsets can be appreciated by considering 

7 the lookup or time correlation table shown in Figure 9. For each video representation (skim 

8 video, fast moving storyboard, slow moving storyboard, and full video), each frame in that video 

9 representation is listed in the appropriate column using the SMPTE time format (Society of 

10 Motion Picture and Television, see http://www.smpte.org), in which hours, minutes, seconds, 

1 1 and frame number are presented as hh:mm:ss:ff. For each of the views, a frame represents 

12 approximately 1/30 second. The rows are arranged so that each entry in a row corresponds to the 

13 same point in chronological time, e.g., each of the 4 views within each row in Figure 9 would 

14 show the same picture. The time correlation table may be computed from analyzing the 

15 corresponding media streams, or may be partially or manually entered into the server. 

16 The various video representations are tied together, so that 60 seconds of viewing time 

1 7 within the slow moving storyboard view correspond to 1 5 seconds of viewing time within the 

1 8 fast moving storyboard view and 30 seconds of viewing time within the full video view. The fast 

19 moving storyboard view by its very nature is a condensed version of the full video, and 

20 consequently, the total number of frames within a fast moving storyboard is less than the total 

21 number of frames of the corresponding full video. The opposite holds true for the slow moving 

22 storyboard. A skim video representation, on the other hand, reflects only segments of the full 

23 video, and hence, there are frames in the full video which do not correspond to frames in the 

20 
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1 skim video representation, as indicated by the dashes in Figure 9. In Figure 9, the skim video is 

2 seen to begin at a point corresponding to 5 seconds into the full video. The skim video continues 

3 for 5 seconds, and then picks up again at a point corresponding to 30 seconds into the full video. 

4 If the user decides to move from the slow moving storyboard view to the full video view 

5 after viewing the slow moving storyboard for 10 seconds, the lookup table in Figure 9 provides 

6 the corresponding total viewing time from the start of the full video required to bring the user to 

7 this point in the story (5 seconds). A time correlation table may also include information on 

8 other kinds of media streams such as the ones disclosed herein, e.g., audio and closed caption 

9 text. 

As an alternative to the lookup table just described, each view (or more generally, each 

\y 
CO 

ry 1 1 media stream) may be associated with its own table, such as those shown in Figure 10, in which 

ru 

£3 12 separate lookup tables are shown for the fast moving storyboard view and the full video view, 

[y 

13 respectively. Each table has an index number in the left hand column, which is associated with 



u 

l 3 



14 the viewing time as given by the SMPTE format. The index numbers are chosen such that they 

1 5 allow the different representations to be correlated in a synchronized way, thereby providing a 

16 numerical link between different views. The indices thus allow one representation to be mapped 

1 7 into (correlated with) another representation. Within the full video view, n may simply be an 

1 8 integer multiple of the frame number up to that point, as in Figure 1 0. In this example, 30 

1 9 seconds of viewing time within the full video view (00:00:30:00), is given by the index 1 800, 

20 which is seen to correspond to 15 seconds of viewing time within the fast moving storyboard 

21 view (00:00:15:00). 

22 Devices such as a PDA or a cellular device might require a stream having different 

23 characteristics (e.g., the stream may have a different frame-rate, different duration, different 

21 



AtSj 

vD 

n 



ri i 



in 



0 

ru 



IBM 920010062US1 

1 audio speedup ratio), and so the offset calculation might be correspondingly different, depending 

2 on the particular device that is used by a user. Also, the methods described herein for calculating 

3 the time offsets between different media streams may be extended to the case in which several 

4 views are displayed on a computer screen simultaneously. 

5 Implementations have been described for switching from a first media stream to a second 

6 media stream in which the viewing in the second media stream actually begins at a point which 

7 just precedes (in the chronological sense) the point at which the first media stream was exited. 

8 Such a method nevertheless entails a synchronization methodology in which the entry point in 

9 the second media stream is determined by and corresponds to (in the chronological sense) the 

10 exit point in the first media stream. This can be done by first computing the corresponding 

1 1 offset, and then decreasing this offset ("going back in time") by a fixed amount, e.g., three 



S3 12 seconds, or by starting the playback of the second media stream at the beginning of the shot 
W 



1 3 containing the computed offset point. The respective positions of the beginning of the shots may 

14 be precomputed and stored as a table, e.g., stored on the server. 

15 The synchronization calculations themselves can be done at either the user (client) side or 

C3 

f=£ 16 the provider (server) side. In either case, the timing details are handled in such a way that the 

17 user experiences a coherent story. For example, the video server 198 can keep a log of the 

1 8 different video streams available to the user and their corresponding lookup tables, thereby 

19 providing the user the ability to jump from a view in one video stream to the corresponding view 

20 in another video stream. Alternatively, lookup tables may be embedded in the data for the 

2 1 browser (e.g., in its web page). If an HTTP server is used, as in many of the preferred 

22 implementations described herein, the timing calculations may also be performed on the 

23 application server 1 86. Timing calculations may also be performed on a server other than the 
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1 video server 1 98 or application server 1 86, e.g., on a intermediate or proxy server (not shown). It 

2 will be clear to those skilled in the art that time correlation tables and time computations require 

3 a small amount of storage and basic computation means, which can be implemented in various 

4 ways. 

5 Further, synchronization data like that shown in Figure 10 may be streamed with the 

6 media stream itself to facilitate the necessary calculations. In this case, it may be advantageous 

7 for the video server 198 to continually update the lookup table data as the video corresponding to 

8 the currently played media stream is viewed. This can be accomplished through the use of an 

9 additional data track dedicated to streaming lookup table data: such data may then be used in 

p 

, % q 1 0 calculations related to one or more media streams, e.g., moving storyboard, audio, closed caption 
CO 

Ty 11 text, etc. 

ry 

f 3 1 2 In a preferred implementation, the timing data which facilitates switching between 

ly 
in 

*" 13 two or more media streams can be represented and delivered as part of a standard for multimedia 

b 

hj 14 by adding the necessary data structures to the standard. For example, using SMIL 2.0, an author 

ry 

V-j 1 5 can describe the temporal behavior of a multimedia presentation, associate hyperlinks with media 
Q 

f * 1 6 objects, and describe the layout of the presentation on a screen. The SMIL format would 

1 7 therefore be an appropriate placeholder for information about the time-relationships between 

18 two or more related media streams, such as tables and formulas. Similarly, the MPEG-7 standard 

19 under development permits content based indexing and access to multimedia streams. MPEG-7 

20 might include timing data to support synchronized switching between related media streams. 

2 1 (For details on SMIL, see Synchronized Multimedia Integration Language (SMIL 2.0) 

22 Specification, W3C Working Draft 01 March 2001 at http://www.w3.org/TR/smil20/) 
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In a preferred embodiment of the invention, there is provided media encoded with 
executable program code to effect any of the methods described herein. These media may 
include a magnetic or optical disk or diskette, for example. In preferred embodiments, this 
program code may be read by a digital processing apparatus such as a computer for performing 
any one or more of the methods disclosed herein. 
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